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Charter middle schools popular enough to 
hold admissions lotteries are, on average, no 
more successful than nearby traditional public 
schools in boosting student achievement, 
behavior, and attendance. However, charter 
schools vary widely; some are more effective 
and some are less effective than nearby 
traditional public schools. Those located 
in large urban areas and those serving 
disadvantaged students are the most successful. 
Overall, parents and students who win entry 
into a charter are more likely to be satisfied 
with their school than those who do not win 
and must attend another school. 



The policy context 

Charter schools are an important and growing 
component of the public school system in the 
United States. There are now more than 5,000 
of these schools, which first appeared in the 
early 1990s, and they are expected to play a role 
in federal school improvement. However, there 
remains considerable debate as to whether, how, 
and under what circumstances charter schools 
improve the outcomes of students who attend 
them. 

The study approach 

This study — the first large-scale randomized trial 
of the effectiveness of charter schools in multiple 
states and types of communities — was conducted 
with 2,330 students who applied to 36 charter 
middle schools that held lotteries for admission. 
The study was funded by the Institute of Education 
Sciences (IES) and conducted by Mathematica 
Policy Research and its partners. 

The study focused on students who applied to 
charter schools that met three criteria: 



• They were middle schools, with entry usually 
in grades 5 to 8. Studying middle schools 
allowed researchers to use state-mandated 
achievement test results in this study. 

• The schools were open 2 years or more. Thus, 
the results would not be influenced by the 
challenges of start-up schools, which are less 
stable in their operations. 

• They had substantially more eligible students 
than they could admit. Choosing schools with 
lotteries created an ideal study sample — two 
statistically similar groups of students, one 
offered admission to the charter they applied to 
and one not offered admission. 

The charter middle schools that were eligible and 
agreed to participate in the study were somewhat 
different from other charter middle schools in 
the United States. Participating charter middle 
schools served more advantaged students than the 
typical charter middle school. For example, the 
study charter schools had smaller percentages of 
students who were eligible for free or reduced- 
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price school meals (44 percent versus 62 percent the same state, so that the average score for all 

nationally) and smaller percentages of students students in the same state and grade is set to 0, 

who scored below proficiency on their state and about two-thirds of students have scores that 

assessments at the time they applied to the charter range from -1.0 to +1.0 (which represents one 

school (34 percent versus 49 percent, in math). In standard deviation). The study also used school 

addition, fewer African American students attended records and student and parent surveys to examine 
the study schools than attended typical charter other outcomes — including student effort in 

schools (16 percent versus 29 percent). school, behavior, and attitudes, as well as parental 

involvement and satisfaction. 



In each charter school, impacts were estimated by 
comparing average achievement outcomes among 
lottery winners with those of lottery losers over 
the 2 years following the lottery, controlling for 
students’ background characteristics. Researchers 
estimated overall impacts by averaging across 
all sites. They also examined which school 
characteristics could be possible predictors of 
higher or lower impacts on students’ test scores in 
reading and math. 

The primary outcome measure was student 
performance on state math and reading tests. 
Because the tests varied by grade and from state to 
state, the scores were converted to a comparable 
scale. The scale is based on how each student’s 
score measured up to all other students within 



Figure 1. Average Year 2 test scores of charter lottery 
winners and losers are not significantly 
different 
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NOTE: Test scores shown here are based on students’ performance 
on state assessments across multiple states. These scores were 
standardized using a scale in which the average score for all students 
in the same state and grade is set to 0, and about 68 percent of all 
students have scores that range from -1.0 to +1.0 (which represents 
one standard deviation). 



The findings 

• On average, study charter schools did not have 
a statistically significant impact on student 
achievement. However, the averages mask wide 
variation across the charter schools in how 
well their lottery winners performed compared 
to the lottery losers, who typically went back 
to their neighborhood schools. Based on the 
comparable scale, math achievement impacts 
ranged from a minus .78 for one charter to a 
plus .53 for another. (There also is statistically 
significant variation in reading impacts, 
although not as wide). “Impacts” represent 

the difference between the charter school 
and the other schooling options available to 
students. Charter schools with high average 
achievement may have negative achievement 
impacts if the other nearby schools provide 
more benefit to students. Similarly, charter 
schools with low average test scores may have 
higher impacts because they provide a better 
educational experience for their students than 
the neighborhood schools. 

• Study charter schools did not significantly 
affect most other outcomes examined, except 
for parent and student satisfaction. These 
outcomes included absences, suspensions, and 
other measures of student performance, as well 
as survey-based measures of student effort in 
school, student well-being, student behavior 
and attitudes, and parental involvement. 
However, lottery winners and their parents 
were more satisfied with their schools. For 
instance, lottery winners were 12 percentage 
points more likely to report that they liked 
school a lot than were lottery losers. Parents 

of lottery winners were 33 percentage points 
more likely to rate their child’s school as 
“excellent” than were parents of lottery losers. 
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Figure 2. Study charter schools show wide variation on Year 2 math score impact 
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NOTE: Each bar represents a study charter school. The red bars show a negative impact on math scores, while the blue bars represent a positive impact. 
Not all of these impacts are statistically significant. 



Study charter schools were more effective for 
lower income and lower achieving students 
and less effective for higher income and higher 
achieving students. 

On average, two subgroups of lottery 
winners — those with initial low test scores and 
those from low-income families — benefited 
academically from admission to charter schools 
(in math but not reading) while their more 
advantaged counterparts did not (lottery 
winners had lower achievement than lottery 
losers in both reading and math). However, 
there were no significant differences in charter 
school impacts for other student subgroups — 
such as those defined by race, ethnicity, and 
gender. 

The variation in student achievement impacts 
among charter schools may be related to certain 
school characteristics. Charter schools in large 
urban areas, those serving more lower income 
or more lower achieving students produced 
positive impacts on student math scores 
relative to other nearby school options. Charter 
schools outside of large urban areas, those 
serving fewer low-income students, and those 



Figure 3. Impacts on Year 2 test scores vary significantly 
by family income status 



Estimated impact 



0.20 



0.17 



0.10 



0.00 



0.05 



- 0.10 



- 0.20 



- 0.12 

- 0.14 



Reading Math 



I I Higher income d Lower income 



NOTE: The estimated impact on reading test scores is statistically 
significant for students from higher income families; in math, it is 
statistically significant for students from both higher and lower income 
families. The difference between the impacts for the lower income and 
higher income groups also is statistically significant in both math and 
reading. 
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serving higher achieving students had negative 
impacts on both reading and math scores. Less 
negative impacts were found in smaller charter 
schools and those more likely to use ability 
grouping. 

The study in context with other charter 
school research 

The Evaluation of Charter School Impacts adds to 
the growing body of evidence on this important 
topic. Although the literature on charter school 
impacts covers a broad range of charter schools 
and methodological strategies, there remain 
gaps in the research that this study was designed 
to address. The study used an experimental 
lottery-based methodology, but — unlike previous 
experimental studies — included charter schools in 
multiple states and in suburban, rural, and urban 
areas. Most previous studies of charter school 
impacts focused almost entirely on impacts on test 
scores and provided little evidence on the impacts 
on other student outcomes. Common themes 
include the following: 

• Studies that covered a wide span of states 
and/or districts, like this study, also found 
that charter schools did not positively impact 
student achievement on average. 

• Previous lottery-based studies — which 
focused on charter schools in large urban 
areas, primarily serving large populations of 
disadvantaged students — have found positive 



impacts. Such findings are consistent with the 
findings of this new study. 

• A key finding from this study — that individual 
charter schools varied in their impacts on 
students’ reading and math achievement — also 
was found in several other studies. 

• While this study found significantly 
different impacts for students from different 
socioeconomic backgrounds, findings from 
previous charter school studies concerning 
variation in charter school impacts across 
student subgroups have been mixed. 

Looking ahead 

This study was not able to determine why some 
charter schools — particularly those in urban areas, 
serving more disadvantaged students — appear 
to be more effective than the traditional public 
schools these students would have attended if 
they had not enrolled in the charter lottery. Nor 
could the study appropriately investigate why 
other charter schools, serving more advantaged 
students, appear to provide less academic benefit 
than the nearby alternatives. Studies focused on 
better understanding these patterns would provide 
useful information for policymakers and educators 
seeking to improve student achievement through 
the expansion and adaptation of charter schools. 



IES develops these briefs to offer short, accessible summaries of complex technical 
evaluation reports. For the full report with technical details, see: 
http://ies.ed.gov/ncee/pubs/2Q104029 . 

Gleason, P., Clark, M., Tuttle, C., and Dwoyer, E. (2010). The Evaluation of Charter School Impacts 
(NCEE 2010-4029). Washington, DC: U.S. Department of Education, Institute of Education Sciences, 
National Center for Education Evaluation and Regional Assistance. 







NATIONAL CENTER for 

EDUCATION EVALUATION 
and REGIONAL ASSISTANCE 



NCEE 2010-4031 



Institute of Education Sciences 
U.S. Department of Education 




